posted 01-10-2013 03:02 PM         

You know, the real honest-to God scientific red meat.

In the indu$try-friendly half-baked insider-written trust-us-the-data-is-cool self-published meta-analytic survey that's on $ale for a hundred bucks at the industry's house-organ web site?

posted 01-10-2013 03:30 PM            

I think it would be easier to appreciate the value of your questions if you just a tad less dramatic about it. Not as much fun. I know. But there is a danger is missing your point if you continue swinging so wildly.

I get it that you are not completely satisfied with the APA (2011) meta-analysis because it was prompted by the APA.

Nobody, including us, is ever completely satisfied by anything. There is always more to do and more to learn.

Nobody has to trust us because the data are cool. We provided enough evidence that they can check for themselves.

To be sure, we are still stuck relying on the published study data - it is a meta-analysis after all. But part of meta-analysis is to begin to assess - and describe how we assess - the strength and value of those included published studies. That way, readers can judge for themselves based on the content and the details and not just the players and authors.

The conversation continues to degrade into discussion about the players instead of a discussion about the data.

The meta-analysis, on the other hand, is all about the data.

But yes, it would be great to have more independent researchers who had the funding, and time, and motivation, and competence, and objectivity to do another analsysis. What we have done is to describe our analysis in a manner that will make that task much easier and much less expensive for someone to eventually do. They can look at our study, and the NRC (2003) report, and the OTA (1983) report, and the Honts and Peterson (1997) amicus brief, and the other systematic reviews - including those of Abrams and Ansley, and Crewson. They can see how we tried to develop a sensible understanding of the differences and similarities of the results of those. They can decide for themselves if they are completely satisfied or if there is a better way to make better sense of the available evidence.

Having seen our analysis, and the others, they can more easily design a better one.

Nobody - except perhaps you - has tried to hold out the APA meta-analysis as the final answer. Attacking it as an insufficient final answer is silly.

You've made your point about the potential for apologetics - now its time to move on and discuss the content and details of the data and the analysis. To continue to neglect that part would indicate that you are might be interested only in casting stones at persons or the profession, with no real interest in advancing anything.

.02

r

---> more in a bit.

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-10-2013 03:33 PM         

But just for the record...

posted 01-10-2013 06:13 PM            

Some of these concerns are interesting.

quote:

---

I'm concerned when a polygraph "expert," lecturing to college kids, says "The polygraph knows when you lie."

---

I'm concerned when another polygraph expert, lecturing to future lawyers -- and faculty -- at a law school, runs what appears to be a trick stim "test."

I'm concerned that APA leadership openly endorses trickery.

I'm concerned that trickery is taught at APA seminars.[/quote]

Anyone who is not concerned about the use of trickery and manipulation has there head in the sand. It is a vulnerability. We have been criticized for PLCs for a long time. Simply ignoring this quandary is not going to gain us anything. Being annoyed with the criticism will not help. Attempting to BS others will not help in the Internet age.

The solution will be to engage a dialogue that leads to better understanding, and then maintain the dialogue. We can also de-emphasize our dependence on manipulation where possible, and that will help.

------

You continue to misrepresent the issue of APA leadership endorsement – and it continues to distract from the value of your other discussion.

For the record: I expressed only that it is unlikely that the demonstration would be considered a violation of an APA standard. If I am wrong, then I invite correction. Simply name the number of the APA standard that it violates.

Also for the record: I used a CVOS nearly exclusively for a few years, though with no expectation that the countermeasure question would actually detect countermeasures. I also used a known solution number test. At the suggestion of stat in this forum, I also used a blind-number stim with no trickery. I can recall missing the number once – with a mart sex offender who happens to have a Ph.D in linguistics. He still had no trouble failing and confessing. Was he a victim? When I read Kircher's prior demonstration paper (now reprinted in Polygraph) I started to get more uncomfortable with the fact that there is no research supporting the efficacy of any acquaintance test except the known solution number test, so I have been using that exclusively for the past few years.

quote:

---

I'm concerned that the individual who controls the official worldwide polygraph narrative, chiefly defined by the APA's two house organs, says polygraph is as accurate is film mammography. That comparison is ludicrous.

---

It is not ludicrous to describe the published evidence. It is ludicrous to ignore it. It is also not ludicrous to scrutinize the strength ad weakness of the evidence, but it is ludicrous to attack the individuals without looking at the details.

quote:

---

I'm concerned that polygraph apologists insist the "test" is robust, when the reality is it's as fragile as can be.

---

Fact: with the exception of techniques with reported extreme high accuracy (i.e., ~100%) the polygraph test does seem to produce results that are robust (i.e., stable and not significantly different) regardless of the esoteric differences among named techniques. In fact, named techniques may have outlived their usefulness.

Other fact: polygraph examiners have engaged in a two-sided conversation at times, stating on one hand that the test provides extreme high accuracy (robust argument), but also that it is imperative that the examiner is “officially trained” in a particular technique, and must not use the technique as originally trained, or the test is invalid (fragile argument). In reality the profession seems to be outgrowing the fragile argument in recognition of the similarities among techniques are reported accuracy rates despite esoteric differences.

quote:

---

I'm concerned that the process is so fragile that holding off on asking a previously reviewed CQ until after the first chart will somehow skew the "test."

---

The concern is that it could affect the test in unknown ways. It could increase accuracy or decrease accuracy. It could increase FN errors or FP errors. It could increase test specificity at a cost of test sensitivity. It could also increase test specificity at the cost of test specificity. We don't know until we study it. It could make no difference. Do we want to endorse the use of experimental methods when we have access to procedures that are already known to work at certain rates of effectiveness?

quote:

---

I'm concerned that the process is so fragile that reducing things to writing for a statement test -- as pioneered by Stan Abrams in cases of understandably provocative questions -- is regarded as "bad polygraph."

---

“Pioneered” is a nice word but is actually vague and metaphorical. Metaphors are great for making impressions with examinees and others, but not so great for real understanding and knowledge. Pioneers are great. They cut trails across uncharted wilderness and make progress possible for the next generation. Do we want to continue to encourage people to cut new trails across every remaining wilderness? Maybe. Maybe not. Sometimes the era of pioneering comes to an end.

If indirect statement tests are all the same or better than test that employ direct behavioral action verbs then why not do these tests every time for every murder, every sex offense, every assault, every criminal investigation. Perhaps we should no longer ask direct behaviorally descriptive questions of anything. If we are uncomfortable with this suggestion, I think that is a clue that something is wrong in terms of our uncertainty about these methods. There may be a time for these, but to endorse techniques with no evidence is not wise when we have stronger theory and more evidence .

quote:

---

I'm concerned that the LEPET "test" - a multi-dimensional, multi-level, multi-facet, muliple-times-removed statement test of its own -- is regarded as "good polygraph."

---

LEPET is simply a multiple issue screening test. Adding a lot of additional multiples is just drama.

I may not have the current information, but the 2002 document I got from AP shows two series with only one question about a statement – lying on the application form. Of course lying on a federal application form is a punishable behavior, just as filing a false police report. It is not a statement test, it is a test of punishable behavior.

LEPET suitability questions (i.e., are you intentionally withholding any information about...”) are very broad. To me it would seem unwise and naïve to EVER believe that someone has told everything. There is always more – unless the person is actually squeaky clean. My sex offender clients are not squeaky clean. So these questions are not useful – unless we consider it useful that everyone fails. Other questions with more circumspect target structure can provide more useful information and more useful pass/fail results to sex offender risk evaluators and risk managers.

I suspect that the LEPET may be useful to the people that developed it, and may appear less useful to others with different program or agency needs. Those agencies are probably looking for only the cleanest of the clean people. Great when you have 20 applicants with better resumes for every person that fails the test. Other agencies may have different tolerances based on a different a different applicant pool. They may need other/adjusted target questions. For example: any drug use in 3 years vs, withholding any information (because so many people minimize). But changing the questions makes it no longer a LEPET because the only thing that makes it a LEPET and not an MGQT is the LEPET questions. Sometimes we see that people want so much to follow the federal programs that they will pretend to do so even when they do not they call it a LEPET when they are not using LEPET questions.

quote:

---

I'm concerned that there are no independent studies attesting to the scientific validity of PCSOT or LEPET "tests," or even incident-specific polygraphs.

---

We are so desperate that any studies on PCSOT could begin to be helpful. OK I take that back. Badly designed studies of any type are not helpful. But you get the point. We need studies in this area. Holding our breath for double-blind independently funded field studies will be a long long wait.

So I wonder: does your concern rise to the point of legitimate corrective action in the form of a conscientious decision to stop doing PCSOT, LEPET and incident-specific polygraphs? Or is this just rhetoric?

quote:

---

I'm concerned that using a PCSOT-style model for the domestic violence treatment triangle is rapidly gaining traction as the next ca$h cow for the polygraph "testing" indu$try.

---

We are already getting criticism about this from people in criminal justice.

quote:

---

I'm concerned that the polygraph "test" is inherently biased against honest subjects.

---

The test itself is not biased. It is the way we interpret the test that is sometimes biased. And there are sometimes thoughtful reasons why test interpretation schemes are biased in certain ways. Its just not simple.

What you are really saying is that the test may be balanced (not biased) in favor of test sensitivity and not test specificity. This may or may not be universally correct to assume. But you, in your private practice, are free to balance the test any way you want. You should, however, account for yourself and make a declarative statement of your alpha levels for both deceptive and truthful results. I suppose you can use Matte's published data to do that if you are not satisfied with the meta-analysis.

quote:

---

I'm concerned that the feds don't video record their polygraph "tests."

---

Concern is OK. Judgment may be underinformed. I'm not even sure this is universally true. I think it is a mistake to assume that they have not thought this through, and made what appears to them to be the best program decisions possible at the present time, given the present array of concerns facing them.

quote:

---

I'm concerned when I hear about the methods the FBI used in Higazy's polygraph "test."

---

I'm not qualified to comment on this.

quote:

---

I'm concerned about the secrecy surrounding federal polygraph programs.

---

There is a balance to be achieved between necessary secrecy, accountability/transparency and participation in the broader professional culture. Again, concern is OK. Judgment may may be much less helpful than dialogue and understanding.

quote:

---

I'm concerned when I read that some federal examiners (or contractors) have reservations about the questionable methods used in their polygraph programs.

---

Again, beyond the reach of my knowledge.

quote:

---

I'm concerned that examiner bias is such an important factor that that APA runs lectures to help keep its own in check.

---

The need for continuing education, including in areas of ethics, is not a basis for indictment of any profession.

quote:

---

I'm concerned that the APA is selling (literally) a half-baked "meta-analytic survey" that is devoid of any independent QC, and is representing their beliefs as "science."

---

Already argued to death, but I will reiterate that this statement is laden with drama and misunderstanding. Continuing to bang the drum about already argued potential for apologetics is just argumentative and unproductive. Awaiting discussion of the detail.

quote:

---

I'm concerned anyone who vigorously challenges APA groupthink is labeled as a troll, bigot, anti-polygraph polygraph examiner or charlatan.

---

That is exactly why I recommend less drama. But then that wouldn't prompt the same level of attention. So, I'm trying to stay focused on the important ideas and not much else. Regardless, its actually not fun to watch people get hurt, or to participate in that, or to get hurt. It also concerns me because the value of the argument and discussion gets lost when people personally to others personalities.

quote:

---

I'm concerned that the APA, for all of its claims, bravado and bluster, is sh*t-scared of accepting the A-P countermeasure challenge, or even doing something along those line in house.

---

AP countermeasure is a stacked deck. Play on dude. The discussion about an in-house exercise is much more interesting.

quote:

---

I'm concerned that the American Psychological Association says polygraph "testing" is hokum.

---

1. Drama. 2. Do we accept them as the final authority? 3. Do we think their conclusion was based on the right information. 4) Do we think they really treated us fairly, and 5) is their position understandable given the outrageous published results they observed at the time?

quote:

---

I'm concerned that the American Medical Association says polygraph "testing" is hokum.

---

Same drama. Same response. I do not believe their conclusion is based on the right information though it may have been understandable at the time. If we accept their conclusion as the final authority then we are done (we are also free to ride the horse like we stole it and engage in any form of charlatanry that we want if believe it is actully hokum and don't decide to get out of the bid'ness).

quote:

---

I'm concerned the the Supreme Court of the United States, in a majority opinion, says polygraph testing is unreliable.

---

Again, is this conclusion based on the right information? If we are hoping to maintain a dialogue and achieve a different conclusion some day then it is best not to consider the story as ended yet.

quote:

---

I'm concerned that the only people who believe in polygraph are those with a vested interest in polygraph.

---

I think if you read both the OTA (1983) report and the NRC (2003) report you will actually see a consensus of outside scientific opinion that polygraph does work – despite the fact that they were not impressed with the silly perfect and near-perfect results and the unsatisfactory quality of a lot of polygraph research.

quote:

---

I'm concerned that polygraph apologists are all for "practical polygraph," but circle the wagons when CVSA makes advances.

---

If we engage in the pursuit of scientific polygraph on one hand and oppose CVSA as unscientific on the other then there is no inconsistency. If we decry evidence-based polygraph as science, and emphasize only the “trust me” expert-practice model and also oppose CVSA as un-scientific then that would be inconsistent.

quote:

---

I'm concerned that two different polygraph scoring algorithms, using the same data, give two different results.

---

The real concern here is not the presence of any disagreement, it is the rate of disagreement and the cause of disagreement. There is always more to learn. And, of course, scientists and algorithm developers need accountability too.How often does this happen? What is the cause? Without access to design and development data we won't really know. That is exactly why we made OSS-3 open source and completely accountable. Oddly, for all our emphasis on accountability, we have been way too comfortable with black-box models for which almost nobody can account for.

quote:

---

I'm concerned that polygraph operators, generally speaking, discourage prospective test-takers from researching the "test" prior to taking it.

---

I, for one, simply assume that every sex offender has researched the test. I think it would be naïve to assume anything else.

.02

r

posted 01-11-2013 08:51 AM         

APA leadership is living in a bubble.

Consequently, its collective judgement is impaired.

To the best of my knowledge, no one (other than me) is taking steps to point this out.

In industry, we hire consultants to fully understand our products, services and business models. These outside experts then go to great pains to question a company's methods -- and expose any resultant faults or weaknesses -- before the company brings anything to market in a big way.

Also, in industry, much money is spent on customer beta sites, which enables an industrial entity to test things in the real world, collect feedback, and then refine (or kill off) a product or service before investing more heavily in it.

Getting outside opinions from independent experts is common in industry for a host of reasons, one of which is avoid a form of intoxication that can creep into a narrow-minded organization.

A cite this cautionary tale because the APA is not only overly invested in its own advocacy research, but everyone's getting drunk on it.

As it stands right now, the APA has a very serious credibility program with its meta-analytic survey.

I suggest that the APA hire at least two reputable scientific process consulting firms -- who should remain unknown to each other's activities -- to vet the meta-analytic survey from top to bottom.

Obviously, the consulting firms will have to develop a comprehensive understanding of polygraph in all of its dimensions.

They must be true outsiders. Untainted.

If the scientific process consultants' reports line up with the APA's findings, then the APA will have a very powerful adjunct body of work to further support its agenda.

On the other hand, if the scientific process consultants' reports finds serious (or possibly fatal) flaws in the APA's meta-analytic survey, then steps can be taken to minimize further damage.

Industrial entities sometimes do quite well for a while living in a bubble. But when they crash, they crash hard.

An independent scientific process review can save the APA from the shock and embarrassment of a similar fate.

posted 01-11-2013 11:07 AM            

I want to sincerely thank you for these more reasoned and rational criticisms and suggestions in the above posting.

Everyone should read this carefully.

Dan is asking us not to get overly invested in APA research. He's right.

He suggests two outside expert reviewers. Of course that would be great. Anyone know who has the chops to do that? Of course, the publication itself invites anyone to read and criticize it, and we discussed an number of these limitations within the report.

I don't think I'm ready to jump right on board with the credibility problem. We've had that problem all along. Little by slowly, we will rise above and overcome that if we do good realistic work. If we publish silly things we will not. There is nothing wrong with an industry or profession studying itself. Of course, it is always better if outsiders study it too. Neglecting to study ourselves would be... negligence.

Publishing what we think we know, making ourselves accountable, and opening the avenues for others to examine and critique our analysis will eventually improve the situation with the credibility problem - if we do things right. If not, our credibility problem will be worsened.

BTW, We do already have outsider review of the polygraph ---> the NRC (2003) report. But we found that it did not satisfy all of our internal questions, such as the question of validated polygraph techniques. So, it had to be done. APA leadership and membership wanted to know which techniques are valid.

But it is important to remember that the meta-analysis is not intended to be a final answer or even to stand alone in answer to questions about validity or accuracy. That is why we also discussed the other systematic reviews, including the OTA (1983) and NRC (2003) reports and others.

Regarding the bubble. We've heard this before too, and it is not completely incorrect. We were told that we have not kept pace with other sciences, and that we use alice-in-wonderland concepts and terms. Certainly this is a reflection of APA leadership, and I am culpable to the degree that I am part of APA leadership today. But I think things are also capable of changing, and challenges like yours highlight the potential hazards.

Your caution is worth noting. Are we facing a crash? Possibly? Can we avoid it? Possibly? But we won't avoid it if we do nothing. Part of doing something is studying ourselves and our own work and attempting to account for ourselves. Another part, as you describe, is inviting outside criticism. Of course, another part is the publication of enough detail that those outside critics are empowered to actually check the facts. I think we have done that.

So again, I don't think the APA meta-analysis is intended to be a final answer. I don't think it is Intended to stand alone to answer questions about validity. I think we already know what an outside review will find. You can find it in the NRC (2003) report. That does not deflate the value of another outside review.

There are, of course, dangers with external review. If we pay for it and they support us then people will say we bought the result. If they don't support us we'd have to decide what to do with it. If someone else pays for it and they are critical we will have an impulse to assume they were secretly against us. If they support us, others will assume they were covertly biased in favor of us. Just look at all the painful reactions to the NRC and OTA reports. Were they biased against us? They were very critical. They also ended up agree that the polygraph seems to work. Now that the pain has subsided we find there is something to learn from their work.

The way to stop the madness is to recognize that we have to do our part, and - as you suggest - expose and invite others to look and critique from the outside. The details are now more available than in the past.

Again, thanks for your persistence on this.

I do think it is important that people not get drunk on this information. Its just another look at what we think we know.

If we continue to do the right work, then we will know more in the future. If we don't then we will not. If we do nothing and wait for some beneficent objective expert committee to come along then we will probably wait until it is too late. If we don't publish and expose ourselves to external review and criticism then we will become cloistered and out of sync with other professions, and we will not withstand external critique.

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-11-2013 03:08 PM            

Your response did offer me the opportunity to see something that I had only previously read about, though: Godwin's law. In essence, Godwin observed many years ago that the longer the debate and hotter the topic, the likelihood of someone invoking the Nazis approaches certainty. Now I can say I've seen it with my own eyes.

Don't take this stuff personally, Dan. Let's just debate the issues.

posted 01-11-2013 06:03 PM         

It won't work.

Why? Because in your case, the shoe -- or perhaps I should say jackboot -- fits alarmingly well.

I dare say that topic deserves its own thread.

But for now, Herr Minister, let's take the focus off of me and put it back where it belongs:

On your indu$try-friendly half-baked insider-written scientifically vacant trust-us-the-data-is-cool self-published meta-analytic survey that you are $elling for a hundred bucks a pop (!) on your propagandist house-organ web site.

posted 01-11-2013 06:52 PM            

what is the goal here?